Papers with probing analyses
On the Complementarity between Pre-Training and Random-Initialization for Resource-Rich Machine Translation (2022.coling-1)
Copied to clipboard
| Challenge: | Pre-Training (PT) of text representations has been successfully applied to low-resource Neural Machine Translation (NMT) however, it often fails to achieve notable gains on resource-rich NMT on par with its Random-Initialization (RI) counterpart. |
| Approach: | They propose to combine pre-training and random-initialization techniques to achieve significant improvements in NMT. |
| Outcome: | The proposed model fusion algorithm can achieve significant improvements on two resource-rich translation benchmarks. |
Homophone Disambiguation Reveals Patterns of Context Mixing in Speech Transformers (2023.emnlp-main)
Copied to clipboard
| Challenge: | 'context mixing' is a feature of Transformers that is used to build up representations of acoustic and linguistic structure in speech models. |
| Approach: | They propose to use a French spelling quirk to probe context mixing in speech models to find out how to translate spoken words into written equivalents. |
| Outcome: | The proposed model incorporates cues to identify correct transcription, whereas encoder-decoder models relegate task to decoder modules. |